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ABSTRACT 

This paper was designed to provide background 
information for participants of the "Statewide Evaluation of Programs 
and Services with Students with Disabilities" Policy Forum .scheduled 
for August 30 and 31, 1994. The paper provides information regarding 
program evaluation issues, to stimulate discussion and to give 
examples of resources available to assist State education agencies in 
planning and implementing program evaluation procedures as required 
annually by Federal law. Twenty-eight documents and reports are 
abstracted to provide specific perspectives and recommendations 
regarding the range of special education program evaluation issues 
and challenges. The reports include, among others, products, of the 
National Center on Educational Outcomes, the National Center for 
Policy Options in Special Education, and Project FORUM of the 
National Association of State Directors of Special Education. 
(JDD) 
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FOREWORD 



This paper is done under Project FORUM, a contract funded by the Office of Special 
Education Programs at the U.S. Department of Education and located at the National Association 
of State Directors of Special Education (NASDSE). Project FORUM carries out a variety of 
activities that provide information needed for program improvement, and promote the utilization 
of research data and other information for improving outcomes for students with disabilities. The 
Project also provides technical assistance and information on emerging issues, and convenes small 
work groups to gather expert input, obtain feedback, and develop conceptual frameworks related 
to critical topics in special education. 

Federal law requires that each State provide for procedures for evaluation at least annually 
of the effectiveness of programs and services for children with disabilities. With the passage of 
GOALS 2000 and on-going statewide education reform efforts, special education program 
evaluation is a dynamic and complex issue and process. The purpose of this paper is to provide 
background information for participants of the Statewide Evaluation of Programs and Services 
with Students v/ith Disabilities Policy Forum scheduled for August 30-31, 1994. It was 
undertaken as part of Project FORUM's work during the second year of the contract. This paper 
provides information to Forum participants regarding program evaluation issues and perspectives 
of other authors/researchers. It is intended to stimulate Forum discussion and to give examples 
of resources available to assist State Education Agencies in planning and implementing statewide 
program evaluation procedures. 
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Background 

The Individuals with Disabilities Education Act (IDEA) requires each State Plan to 
"provide for procedures for evaluation at least annually of the effectiveness of programs in 
meeting the educational needs of children -with disabilities (including evaluation of individualized 
education programs).... " (Section 1413(a)(l 1)). In addition, Section 435(b)(4) of the General 
Education Provisions Act (GEPA) calls for the State to evaluate the effectiveness of covered 
programs (including the IDEA) at least every three years. 

In addition to Federal requirements, the education reform movement of (he past ten years 
has resulted in a shift from documenting the process of educating students to demonstrating 
positive outcomes. Legislators, governors, State and local boards of education, and the public all 
are demanding more program accountability within the context of education reform. Financial 
constraints have also prompted an additional need to ensure maximum program effectiveness. 
State Improvement Plans that will be required under the provisions of the newly-passed GOALS 
2000, Educate America Act, must have procedures for improved governance and management. 
Educators at each level of governance continue to document educational effectiveness for all 
students, including those with disabilities, with many different kinds of accountability practices. 

Accountability and program improvement are primary reasons for implementing statewide 
evaluations of educational programs for all students, including those with disabilities. An overall 
assumption of this paper is that statewide educational program evaluation systems must include 
evaluation of program outcomes and impact upon students with disabilities. In addition to 
information gathered for all students, statewide program evaluation systems should have the 
capability of gathering information specific to programs and services provided for students with 
disabilities (e.g., post school employment, impact of specific types of intervention, etc.). 



Purpose of this Document 

As stated in the foreword of this paper, the purpose of this document is to provide 
background information for participants of the upcoming program evaluation Forum, August 30- 
31, 1994, Arlington, VA. It is meant to serve as a Forum resource and to stimulate discussion 
at this upcoming event. Twenty-eight documents and reports related to special education program 
evaluation have been abstracted to provide specific perspectives, information, and 
recommendations regarding the range of special education program evaluation issues and 
challenges. The abstracts are in alphabetical order by author. This document also provides 
information in a single source regarding work on the topic of program evaluation by policy 
analysts and researchers including those from some of the projects that have been supported by 
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the Office of Special Education Programs (OSEP), U. S. Department of Education, incoiding the 
National Center on Educational Outcomes and the National Center for Policy Options in Special 
Education, and Project FORUM of the National Association of State Directors of Special 
Education. The level of specificity of summary information within this paper varies for specific 
documents reviewed and summarized. This varying level of content specificity relates to the 
nature of the Program Evaluation Forum agenda and is intended to provide specific resource 
information and to stimulate discussions among the Forum participants. 



Summary of Selected Program Evaluation 

Ahearn, E. 1992. An analysis of state compliance 
Association of State Directors of Special 



Documents and Reports 

monitoring practices, Alexandria, VA: National 
Education, Project FORUM. 



♦ Current monitoring practices were analyzed from several sources: discussions with 
monitoring personnel in selected States, responses to a survey conducted by the Regional 
Resource Centers in June, 1992, input received at meetings and conferences, and policy 
and other documents. 



♦ The author found that many States are examining their monitoring practices and 
considering changes to make the process less confrontive. 

♦ The topic of using information about student outcomes as a part of compliance 
monitoring is receiving increased attention in the States. 

♦ A number of States are using focused/targeted monitoring. 

♦ A lack of correlation between compliance monitoring and program effectiveness was 
noted by participants involved in this study. 

Berkowitz, S. 1993. A guide to conducting stakeholder-based evaluations for state agency/federal 
evaluation studies projects. Rockville, MD: Westat, Inc. 

♦ This paper provides an overview of stakeholder based approaches to conducting 
evaluation studies, and presents key considerations that should be involved in using 
stakeholder-based evaluations. 

♦ Stakeholder-based evaluation involves active participation in the evaluation process by 
stakeholder groups whose interests are affected by the program being evaluated or whose 
decisions can influence the direction of the program. 

♦ The impetus for this type of evaluation has been that evaluation results were generally 
not being used. Evaluation research often has too narrow a focus, unrealistic outcome 
standards, and lacks responsiveness to the practical and policymaking needs of the 
various and often disparate constituencies involved. 
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♦ 



Stakeholder-based evaluation has emerged to help promote greater use of study results, 
to get constituencies more invested in the process and products of research, and to make 
evaluations more responsive to diverse needs. Stakeholder-based approaches work 
especially well for evaluations that seek to integrate and reconcile diverse perspectives 
on a given issue or program. The role of stakeholders include framing the evaluation 
questions or at least reacting to/revising evaluation questions. 

Using stakeholder-based evaluation involves making decisions on a number of issues, 
including determining availability of stakeholders, selecting the stakeholders, deciding 
how and when stakeholder involvement can most effectively be used, finding appropriate 
and flexible organizational arrangements for conducting the evaluation, developing a 
workable role for the evaluator, and establishing clear expectations among stakeholders 
and project personnel regarding the roles to be played in the evaluation. 

There are a variety of ways in which stakeholders may be involved in developing and 
assessing the evaluation design and methods; e.g., participating in the creation of the 
design and providing substantive direction to the evaluation design while the researchers 
handle the technical aspects. 

Stakeholder feedback in instrument development and revision is particularly important 
in validating the content of instruments, and to ensure that the appropriate topics are 
included so that the face validity of the items is high. 

The following strengths and weaknesses of stakeholder evaluation are also provided 
within this report: 

It results in greater stakeholder ownership of the evaluation process and results. 

• There is shared accountability. 

• Use of stakeholder-based evaluation may entail loss of control over the evaluation 
design by the evaluator which, in turn, may mean diminished methodological rigor 
of the evaluation. 

• There is a trade-off between a tight design and greater participation. Conflicts can 
occur between a top down approach and a participatory paradigm. 

• In stakeholder-based evaluation, active participation in the evaluation by 
stakeholders with diverse interests makes for articulation of a greater variety of 
views than is typical in most evaluations, but it may also be difficult to effectively 
reconcile stakeholder differences. 

Some differences of opinion among stakeholders are to be expected, but too much 
disagreement, especially when it is not productively channeled, can be debilitating. 

• In stakeholder-based evaluation, there is less separation between process and 
product— and more interactive feedback over the course of the evaluation than is 
typically the case in most evaluations. 

People who are heavily task and product oriented may become impatient with the 
amount of attention and energy that gets devoted to process. 
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A stakeholder-based approach is generally not the most efficient way of finding 
answers to the evaluation questions because it is time consuming. 

Brauen, M., O'Reilly, F., & Moore, M. 1994. Issues and Options in Outcome-Based Accountability for 
Students With Disabilities. College Park, MD: University of Maryland 

♦ The purpose of this document is to create awareness of the issues related to including 
students with disabilities in outcome-based accountability systems and to provide options 
for implementing such systems. 

♦ A framework is provided for creating an outcome-based accountability system that 
includes students with disabilities. The focus of the document is outcome-based 
accountability at the local level. 



♦ 



♦ 



Issues and options are presented according to four critical decisions that must be made 
in creating an outcome-based accountability system: 

selecting outcomes for all educational programs; 
establishing performance standards; 
identifying assessment strategies; and 
identifying accountable parties. 

Following the issues options, there is a series of steps provided that are necessary to put 
an outcomes-based accountability system into practice. 

Within this document, the term "outcome-based" emphasizes the central role of student 
outcomes in a system of accountability rather than compliance with procedures and 
practices. Accountability means a systematic method to ensure those inside and outside 
the educational system— both schools and students are moving toward desired goals. 

Outcome-based accountability requires assessment of outcomes and routine reporting of 
student performance using agreed-upon outcomes. 

♦ Two major approaches to consequences within accountability systems are discussed: 
those that involve the automatic imposition of sanctions or rewards and those that rely 
on the public disclosure of results. 

DeStefano, L. 1993. The effects of standards and assessment on students in special education. Synthesis 
Report 10. Minneapolis, MN: University of Minnesota, National Center on Educational 
Outcomes. 

♦ This paper provides an overview of the demographic characteristics of students in special 
programs, outcomes of schooling, and the nature of special education programs and 
services. 



♦ 



Evaluation Policy Forum Background Paper p , 

Project FORUM at NASDSE Final Report - August n%4 



ERIC 



♦ Given the high incidence of risk factors and poor outcomes associated with education, 
students with disabilities need the benefits of an accountability-based school reform 
focused on educational outcomes. 

♦ Since the majority of instruction for students with disabilities takes place in regular 
classes, regular educators should assume responsibility for their learning. 

♦ Effective schools have a shared vision. Outcome-based accountability for all students is 
difficult to achieve unless programs and activities have a common set of goals for all 
students and a shared vision of the standards. 

♦ While a procedural focus on safeguards and compliance in special education has had its 
benefits, outcome-based reforms require the system to go beyond— to judge the schools 
and students in terms of outcomes by applying rewards and sanctions based on these 
outcomes. 

♦ A dilemma exits between the conflicting priorities of reaching higher school 
performance scores and the emphasis on inclusion of students having difficulty 
developing the skills assessed. 

♦ Some have seen IEPs as an alternative to participating in statewide testing for 
accountability purposes. However, the IEP is designed to monitor progress toward the 
achievement of short term objectives and annual goals rather than long term progress. 
The IEP is not valid for program monitoring or individual student assessment without 
a consistent framework for tying IEPs to goals and standards (Olsen & Massanari, 1991). 

♦ Challenges of including special education students in large scale assessment include 
identifying ways or modifying administration procedures to make assessment accessible 
for all students and creating alternate ways of gathering performance information related 
to overall standards and needs of individual students. 

♦ In order to be valid, the measurement, characteristics of an assessment system must be 
reflective of the diversity of students it represents. Multiple data gathering strategies 
should be utilized to accommodate the unique assessment needs. 

♦ Determining the appropriate level of aggregation for reporting assessment data is another 
difficulty in designing an accountability system that represents a diverse student 
population. 

♦ This report includes several strategies for increasing the positive impact of standards and 
assessment for students in special education (adapted from Sawyer, Warren, and 
McLaughlin, 1992): 

Consensus should be made known on the underlying assumptions and framework 
for an outcome-based accountability system that includes students with disabilities. 
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The parties who are being held accountable for the outcomes of special education 
students and all students should be made known. 

• A set of agreed upon outcomes and standards of performance with relevance for all 
students should be developed. 

Appropriate sources and measures of outcome data should be determined. 

• A system should include reporting formats that allow for aggregated and 
desegregated reporting. 

• The unintended consequences of a system of accountability for students in special 
education and other diverse groups should be assessed through action research. 

DeStefano, L. & Wagner, M. 1990. Outcomes assessment in special education: Lessons learned. Urbana- 
Champaign, IL: University of Illinois 

♦ This report provides a synthesis of studies in special education outcome assessment 
conducted as part of the State Agency/Federal Evaluation Studies (SAFES) program. 
The SAFES program was established in 1983 with the passage of the EHA Amendments 
of 1983 (P.L. 98-199), Section 618. The Congressional intent in creating this program 
was to promote cooperative efforts between States and the Federal government in 
generating evaluation information to impact special education programs at the Federal, 
State, and local levels. 



♦ 



♦ 



Although outcome assessments differ widely in design, measures, and data collection 
approaches, they have a common focus on outcomes as individual achievements, 
statuses, or behaviors. 

Assessment of special education outcomes can be broad (e.g., describing current 
employment status of special education graduates at a national, State, or local level) or 
more specific focusing on testing a particular hypothesis or on examining the effects of 
a specific intervention or system change. 

The funded SAFES projects described within this report cover four focus areas: (1) 
evaluating the effectiveness of special education statewide; (2) comparing the 
effectiveness of alternative methods of special education service delivery; (3) assessing 
the impact of secondary programs on post-school outcomes; and (4) investigating the 
impact of minimum competency testing and/or increased graduation requirements. 

This paper also outlines several activities related to the process of assessing outcomes: 

• Collaborative planning increases stakeholder's support and use of outcome 
assessments. 

• Needs assessment should be carried out in the early stages of planning for outcome 
assessment and should include all stakeholder groups. 

• A conceptual framework is critical to the success of an assessment in that it 
provides a structure for understanding, interpreting, and acting to influence 
outcomes. 
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• A conceptual framework should include both proximal and distal outcomes > ey 
independent variables that are expected to influence outcomes, and indications of 
the expected relationships among them. 

• A conceptual framework can be generic or developed expressly for a specific 
purpose. 

Gonzales, P. 1994. Effectiveness, compliance, accountability, and program improvement: Perspectives 
. on the evaluation requirement. Alexandria, VA: National Association of State Directors of 
Special Education, Project FORUM. 

♦ This document references a recent study by NASDSE (Gonzalez, 1 992) in which it was 
found that the majority of States (94 percent) used compliance monitoring procedures 
as the predominant program evaluation tool and that program evaluation and compliance 
monitoring tend to be viewed by State Education Agencies as the same. 

♦ This paper discusses four sub-topics related to program evaluation of special education- 
suggested linkages and distinctions between effectiveness evaluation and implementation 
evaluation; linkages between compliance monitoring and the concepts of program 
improvement and accountability; utilization of evaluation results; and recommendations 
tor a re-conceptuahzation of the annual evaluation requirement under Part B of IDEA. 

The measurement of outcomes is central to determining the effectiveness at the propram 
or individual level. Outcome evaluation compares actual outcomes with desired outcomes 
(objectives/goals). 

"Implementation evaluation focuses on finding out if the program has all its parts, if the 

IZoTwCX if the program is operating 88 * is suppose t0 be operatin8 " 

♦ Compliance monitoring is one form of implementation evaluation that answers questions 
such as: Was the program implemented as designed?" or "Were the mandated guidelines 
and requirements being met?" Compliance evaluation allows the SEA to identify patterns 
of compliance within and across districts as well as program st-engths and program 
components needing technical assistance. However, the collection of quantitative data 
does not provide information regarding contextual variables, or processes; and it does 
not provide information on where and how to make improvements or the information 
necessary for program replication. 

♦ Process ■evaluation focuses on why various things happen, how the parts of the program 
tit together, and how people perceive the program as a whole (Patton, 1986). 

♦ Component evaluation looks at. a specific component of the program rather than the 
entire program. 



♦ 



♦ 
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♦ The overall purpose of the evaluation and the use of the results must be considered in 
the design of a system of program evaluation in special education. 

♦ Program improvement and accountability are two frequently cited reasons for evaluation. 
They have distinctly different implications for the conduct of program evaluation. 

♦ Program improvement questions include: What are the strengths and weaknesses of the 
program? How d n the program be improved? What is working well and what is not 
working well? What are the reactions of clients, staff, and others to the program? What 
are their perceptions about what should be changed? 

♦ Accountability questions include: Is the program implemented as required? (program 
compliance monitoring) What are the planned and unplanned outcomes? (program 
effectiveness) DeStefano (1992) has nGted that while compliance monitoring and 
evaluation both contribute to accountability, they are not the same thing. 



♦ 



There is significant interest across the country in developing a system for combining 
compliance monitoring and outcome evaluation for accountability purposes. 

♦ This document points out several problems with using outcomes evaluation in the 
context of accountability; e.g., holding an entity accountably for their performance 
suggests a system of rewards and/or punishment; a cutoff point between satisfactory and 
unsatisfactory is needed; and cooperation of participating school districts is needed. 

♦ Evaluation for the purpose of accountability and compliance monitoring serves a 
"signalling" function. 

♦ Decisions at the Federal and State levels need to be made regarding the overall purpose 
of program evaluation in special education — either for accountability or for program 
improvement. 

Gonzales, P. 1993. State procedures for the evaluation of special education program effectiveness. 

Alexandria, VA: National Association of State Directors of Special Education, Project 
FORUM. 

♦ The purpose of this study was to examine the efficacy of program evaluation under the 
Individuals with Disabilities Education Act (IDEA) by describing the methods currently 
used by States to conduct their annual evaluation of program evaluation. 

♦ There are five purposes of evaluation (Wentling, 1989) (1) to convince or gain support; 
(2) to educate or promote understanding; (3) to secure the involvement of key 
individuals; (4) to improve programs; and (5) to demonstrate accountability. 

♦ Most of the States utilize compliance monitoring in the annual evaluation sections of the 
State Plan and Part B Performance Report. Monitoring data sources such as LEA 
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• 



♦ 



♦ 



♦ 



applications, site visits, or fiscal audits provide very limited information upon which to 
evaluate the effectiveness of special education programs and services. 

There is an increase in the number of States that use a two-tiered approach to the annual 
evaluation requirement in which the Local Education Agencies (LEAs) have evaluation 
requirements and the State Education Agency (SEA) has monitoring requirements. 

There is an increase in the number of State, that encourage LEA "self-study" often as 
a pre-monitonng tool or an evaluation tool. 

t?^! inC T e in thC nUmb6r ° f States that P'-° vide techl ^al assistance to LEAs 
to help them evaluate their programs and procedures. 

' f^T^J^T thCy WCre SUpP ° rting shorMerm studies t0 evaluate specific 
i sues projects and/or practices in special education. For example, Kansas funded a 
study that examined the effects of class-wide peer tutoring on at-risk students' 
subsequent placement in special education. Minnesota funded a project to evaSe the 

m t? nT SS - ° f dirCCt IangUage thera Py therapy provided through 
mainstream collaboration. Some States reported that the results were used to influence 
policy decisions, State technical assistance priorities or comprehensive system of 
personnel development (CSPD) plans. y 

♦ The annual evaluation sections of source documents from ten States indicated that 
evaluation reports were required from projects funded by IDEA-B discretionary mon es 
mcluding model projects initiated by LEAs. Evaluation from individual projecL has the 
evSn mflUenCe P ° hCy 8nd t0 Cngender SU PP° rt for P"ject replication aX 

' e^alutlonTof " ^ ^ ^ t0 COnduct or fu » d statewide 

evahiations of special education, including studies that focus on the post-school status 

of graduates and on the effectiveness of transition services. Iowa, for example ha 

funded a 5-year statewide foliow-up study of special education graduates us^g Part B 

discretionary funds. Maryland is supporting a statewide evaluation study of the StaSof 

ransi tion services m schools and community colleges. California State hw cat for 

annual evaluation studies of issues of statewide concern in special education e g pupi 

performance; placement of pupils in the least restrictive environment (L^^he degree 

to which services identified in lEPs are provided; parent and pupi abides toS 

services and processes provided; and program costs. 

♦ The individual education program (IE?) is receiving attention in a number of States for 
use mi program evaluation. The 1EP can provide the ideal tool for identify ng ind vidua 
EP S IT 6 r° meS ^ aSSCSSing Student P roeress - In °' de ^o evaluat e the 

eva uauon by the LEA along with compliance monitoring by the SEA- and 3 
evaluation by the SEA. usually collecting data from a sample of lEPs Examples of 
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Arkansas); and an Evaluation Activity Guidebook for LEAs (Illinois). 

♦ Some States have begun to investigate ways of collecting date on student performance 
olomes for the pulses of evaluating the effectiveness of special education. 

♦ Four States (Indiana Michigan, Oklahoma, South Carolina) specifically mentioned 
^udTcom^ive technical assistance or guidance documents that encourage 
LEAs to use student outcomes in the evaluation activities. Connecticut, Delaware, 
Kentucky and Maryland are in various stages of implementing statewide requirements 
^tx^dll outcome information as a basis for LEA and/or SEA evaluation. 

♦ Three States (Vermont, Utah, and West Virginia) reported a process in place for 
Electing statewide information on student outcomes for evaluation purposes. 

♦ Four States (Connecticut, Delaware, Kentucky, and Maryland) indicated they are in 
various stages of implementing such a process. 

♦ The majority of States (94%) continue to use compliance monitoring procedures as a 
J^^wtion tool Corrective action following compliance monitoring is equated 
S^^JLing program improvements. Program evaluation and compliance 
monitoring frequently are viewed as the same activity. 

♦ There is a need to promote a balance between the human and fiscal resources necessary 
Ir inducting compliance monitoring and program evaluation at all levels of specia 
riu^SnLLn based upon their relative contribution to improving outcomes 
for children and youth with disabilities. 

♦ Technical assistance in the area of program evaluation is needed at the SEA and LEA 
levels. 

» Slate and Federal laws and regulations should avoid restricting the types of program 
ev^ualns 4at 1 be conducted by SEAs and LEAs. In addition, State and Federa aw 
Z TegTations should evolve to reflect changes in the educattonal system as well as 
national standards and community values. 

McLaughlin, J. S. 1988. Special education program evaluation: An overview. Blacksburg, VA: Virginia 
8 Polytechnic Institute and State University, College of Education. 

♦ Evaluation is a process through which evaluators gather information for decision makers. 

♦ Generally, evaluations are conducted to meet the needs of internal or external people 
who are making judgments and/or decisions about the program. 
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♦ Federal and/or Stue regulations call for the evc-uation of special education programs. 



♦ Aims of evaluation include program improvement, sharing of success, and providing a 
tool for advocac> . 

♦ Information generated from an evaluation is usually combined with other information; 
e.g, political, social, economic, and administrative factors. 

♦ There are four f icets to most evaluation efforts: 

• statement of evaluation questions which guide the development of the evaluation 
plan; 

• information gathering (formal or informal); 

• judgement in the interpretation of findings (comparison of the information to some 
standard or expectation; e.g., from legislation, regulations, guidelines, court 
decisions, literature, and professional experience); and 

• the decisions about the program under consideration are; to continue it until the 
standard is met, keep the standard but revise the program, revise the standard, 
terminate the program, or disseminate the program if it consistently meets the 
objectives or standards. 

♦ Several steps to planning evaluation are provided within this document: 

• Getting started - This step involves focusing the research. 

• Describing the program - This step provides information about the program in terms 
of inputs (resources), processes (activities), and outcomes (products or benefits, 
changes in programs or clients). 

• Developing evaluation questions - Evaluation questions link the program design to 
the evaluation design and serve as the vehicle through which needed information is 
provided to the evaluation team. Seven basic questions might be addressed: Are we 
achieving goals as predicted? Did we achieve our goals? Did we cause the changes 
in the target? How is the program perceived? What did the program cost? Is the 
program cost effective? Are students successful after leaving the program? 

• Collecting the information/data - This step involves matching the information/data 
collection to the information needed to answer the evaluation questions. 

• Planning and carrying out the analysis - The evaluation will result in the collection 
of a considerable amount of data from various sources. The purpose of data analysis 
procedures is to reduce raw data to a manageable form to allow for interpretations 
and/or inference with regard to the evaluation questions. The techniques and 
presentation strategies should be shared with audience representatives to determine 
their utility for judgment and decision making tasks. 

• Preparing the report - Reporting closes the evaluation cycle with formal or informal 
reports and varied formats (written, verbal, audio-visual). 
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♦ The author of this paper outlines several strategies that can enhance utilization: 

Evaluators should identify and involve decision makers at the start of the evaluation 
process. 

• Evaluation designs must be feasible. 

The evaluation should be conducted in a legally and ethically responsible way (e.g., 
protection of the rights of those participating). 

Evaluations should be technically sound (e.g., reliable, valid and use objective data 
gathering procedures. Discussions, conclusions, and recommendations must be 
logically related to results and evaluation questions. 

Utilization and dissemination plans should be an integral part of the evaluation from 
the start. 

• The evaluation staff should be cognizant of contextual factors which might Facilitate 
or impede information usage. 

• The evaluation staff should view themselves as change agents. 

• The perceptions of the user will influence the degree to which evaluation results 
have impact on programs. 

• Multiple source/multiple methods broaden the acceptability of evaluations. Different 
data gathering strategies (interview, survey, and/or record review) and different 
sources (parents, teachers, and administrators) can be used to answer the same 
evaluation question. 

• The evaluation should be anchored to specific program elements. 

• Information retrieval and processing systems should be reviewed. 

McLaughlin, M., Pasco, S., Schofield, P., & Warren, S., (University of Maryland); Brauen, M., Heid, 
C, & O'Reilly, F. (Westat, Inc.); and Moore, M. & Waldman, Z. (Consultants). 1993. Issues 
and options in outcomes-based accountability for students with disabilities. College Park, 
MD: University of Maryland, Center for Policy Options in Special Education. 

♦ Issues and options are presented within this document according to four critical decisions 
that must be made in the creation of an outcomes-based accountability system: 

1 . Select outcomes for all educational programs 

2. Establish performance standards 

3. Identify assessment strategies 

4. Identify accountable parties 

♦ This document is intended for use at the district and school levels in designing and 
implementing outcomes-based accountability systems. 

♦ Outcomes-based accountability requires the assessment of outcomes. It also requires that 
student performance on agreed-upon outcomes is routinely reported and that 
consequences follow. 
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♦ Two approaches to consequences are used: those that involve the automatic imposition 
of sanctions or rewards, and those that rely on the public disclosure of results to have 
a consequence. 

♦ The selection of outcomes requires agreement on both long-term and short-term goals 
of instruction and alignment of the curriculum with the established goals and objectives. 

♦ Performance standards are benchmarks against which student performance can be 
compared. They also set the agreed-upon outcome expectations for student performance. 

♦ Sources of outcome data include standardized achievement test results as well as 
curriculum-based assessments; alternative assessments such as portfolios and authentic 
assessment; functional assessments; school records; and student, parent, or employer 
surveys. 

National Association of State Directors of Special Education. 1993. A forum to examine policy and 
practice surrounding the annual evaluation of program effectiveness. Alexandria, VA: 
Author. 

♦ This Policy Forum was convened to suggest strategies that can be implemented within 
the next three years to improve evaluation efforts and orient policymakers, 
administrators, and advocates to the benefits of program evaluation as an integral psxt 
of program improvement. 

♦ Solutions to other barriers restricting the evaluation of program effectiveness were also 
considered, including resource allocation, lack of technical assistance and methodological 
issues. 

♦ An overall assumption of this Policy Forum was that evaluation must be perceived as 
an integral part of decision making, management, and administration. 

♦ In addition, program improvement efforts require systematic evaluation in order to judge 
their effectiveness and provide guidance for future innovation. 

♦ A concern was noted that evaluation efforts can increase paperwork and take more time 
away from children. In addition, there is a lack of consensus as to what effective means 
and how to determine it. 

♦ It is also difficult to evaluate different programs within the context of the total education 
of the child. 

♦ There is a need to systematically collect consumer satisfaction data from the perspective 
of students and families. 
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♦ Interdistrict differences in the categorization of disabilities, placement criteria, and other 
variables confound the assessment of effectiveness across school districts. 

♦ There is a need to link State and local level special education evaluation systems with 
emerging general education evaluation efforts and the inclusion movement. 

♦ There is currently a misalignment between Federal, State, and local needs for evaluation 
information. 

♦ Attempts to make program evaluation systems serve both accountability and program 
improvement often creates conflicting messages and incentives for providing accurate 

. and complete information. 

♦ Several recomme^' ions were made by Forum participants: 

• A series of } apers should be developed and disseminated regarding what special 
education policy and practice can contribute to the school improvement movement 
from special education research on topics such as parent involvement, individualized 
education, instructional strategies, transition, and cross-agency collaboration. 

• The general education system must include information pertaining to all students. 
Ways for specific students to achieve or measure these outcomes, however, may 
vary. 

• There is a need to develop a conceptual model of educational outcomes or 
indicators of effectiveness that is congruent with models used in general education. 

• The two major problems in assessment of outcomes are: When do you include the 
assessment of outcomes, and when do you accommodate outcomes in the evaluative 
process? 

• It is important to identify stakeholder groups and to systematically assess their 
informational needs. A single evaluation will not address all their needs. 

There is a need for the Office of Special Education Programs (OSEP), Office of 
Educational Research and Improvement (OERI), Regional Labs, and Regional 
Resource Centers to convene groups on a regional basis several times a year to 
provide evaluation information and training. 

OSEP should commission a paper that would identify what regulatory and policy 
initiatives would support the program effectiveness, evaluation requirements under 
the Individuals with Disabilities Education Act (IDEA). 

• Therj should be an analysis of the progress funded by the State Agency Federal 
Evaluation Studies program to determine lesfions learned related to the evaluation 
of program effectiveness and to inform State and Federal evaluation personnel of 
alternative approaches to the broader issues associated with program evaluation. 
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National Center on Educational Outcomes. 1994. State special education outcomes: J 993, Minneapolis, 
MN: University of Minnesota. 



♦ This report includes the results of the third national survey of State activities in the 
assessment of educational outcomes for students with disabilities. 

♦ Major findings of this survey include the following: 

• States are attempting to produce better information on the numbers of students with 
disabilities taking part in statewide assessments. 

• More attention is being directed toward guidelines that help define who participates 
in statewide assessments, with the apparent goal of increasing the number of 
students who participate. 

• Guidelines on acceptable testing accommodations and adaptations are being 
advanced by State assessment programs. There is a trend to allow more kinds of 
modifications, in both low and high stakes assessments. 

• States implementing nontraditional forms of assessment continue to retain the same 
approach to including students with disabilities and making accommodations as in 
their traditional assessments; e.g., that is, many students are excluded. 

♦ Twenty-seven States have more detailed accounts of student time in general or special 
education. 

♦ Fourteen States collect data on reasons for student dropout. 

♦ Fourteen States that award multiple diploma types keep track of these at the State level 
for special education students. 

♦ Almost all States that collect participation and exit data use the data in reports for State 
agencies, legislatures, and local and State Education Agencies. 

♦ The primary outcome areas within State assessment activities are achievement, 
vocational skills, and post-school status. 

♦ Forty-four regular States and all of the unique States collect State level achievement data 
that include students with disabilities. 

♦ Thirteen regular States and three unique States collect vocational skills information. 
Enrollment in vocational education and job placement are the most frequently collected 



♦ The most common method of assessment achievement are norm-referenced tests. The 
most frequently used is the Stanford Achievement Test. Two regular States reported 
using portfolio assessments. 



data. 
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National RRC Panel on Indicators and Effectiveness in Special Education, 199 . Effectiveness indicators 
for sp- cial education: A reference tooL Lexington, KY: University of Kentucky, Mid-South 
Regional Resource Center. 

♦ This document was the product of the work of a national task force. 

♦ The document is intended to be a resource for State Education Agency administrators; 
local education agency administrators; teacher groups; and parent groups to move 
beyond compliance to focus on the effectiveness of regular education, special education, 
and related services for students with disabilities. 

♦ The document has three purposes: (1) to reduce redundant efforts in the future; (2) to 
promote common understandings about what characterizes effective education and 
related services for students with disabilities; and (3) to emphasize the integral 
relationship between special education and the regular education process. 

♦ Several major themes are emphasized in the document: 

• Inter-relationships between regular and special education. 

• The importance of active parent involvement and participation. 

• Meaningful implementation of the least restrictive appropriate environment (LRE) 
concept. 

• A focus on the needs, abilities, and interests of individual students. 

• A broad definition of what achievements and outcomes are desirable for students 
with disabilities. 

♦ Effectiveness indicators are organized into six major areas: 

• Philosophy, policies, and procedures 

• Resource allocation 

• Staffing and leadership 

• Parent participation and community and interagency involvement 

• Instruction 

• Program and student outcomes 

♦ Each category of indicators within the major areas includes a rationale statement 
followed by a list of specific indicators. The indicator statements vary in specificity, 
reflecting the variation found in the literature. 

♦ Most indicator statement can be applied to all students. 

♦ Some statements do not apply to all students with disabilities, and are more related to 
one or more subgroups along the special education continuum, for example an emphasis 
on independent living and integration into the community would be a particularly 
relevant outcome criteria for students with more severe disabilities. 
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♦ At the end of each category, literature sources are indicated by author and date; 
published documents and documents produced by a State or local agency are indicated 
by code. A full bibliography is also provided that gives complete citations for all sources 
drawn from the literature, and a companion list of other sources and codes. 

♦ The primary goal behind the Effectiveness indicators product is to encourage program 
review, analysis, and improvement. The indicators provide a basis for assessing 
effectiveness and a structure for organizing evaluation and improvement efforts. 

Olsen, K. & Massanari, C. 1991. Special education program evaluation: What should states consider? 
Lexington, KY: University of Kentucky, Mid-South Regional Resource Center. 

♦ SEAs have many reasons to install and maintain State systems for special education 
program evaluation, regardless of Federal mandates or initiatives for evaluation. Horvath 
(1985) suggested the following reasons: accountability, public relations, program 
improvement, and policy development. 

♦ The following are characteristics of an effective SEA special education evaluation 
system: 

• A clear sense of purpose and philosophy about the development and use of a State 
program evaluation system for special education; 

• State level consensus on the educational outcomes that are valued for students with 
disabilities; 

• Provision of ongoing data on the attainment of valued outcomes; Three questions 
must be addressed in designing a system: 

What data will serve as indicators of outcome achievement? 
What formats and procedures are best for collecting the data? 
How will data be stored for use in decision making? 

• Provision of 'training and technical assistance for locai programs to conduct self- 
evaluations. 

• Provision of ongoing data on the resources, practices, and results of SEA and LEA 
special education roles and functions; 

• Procedures for conducting policy studies on identified issues; 

• Polices, procedures, and plans that ensure data based decision making in the SEA 
about programs; and 

• A consistent SEA practice of reporting to stakeholders on the status of special 
education programs. 

Shriner, J., Kim, D., Thurlow, M., & Ysseldyke, J. 1993. IEPs and standards: What they say for 
students with disabilities. Technical Report 5. Minneapolis, MN: University of Minnesota, 
National Center on Educational Outcomes. 

♦ While the individualized education plan (IEP) has been applauded as an ideal approach 
for all students, it has not been the answer to educational service provision. In fact, it 
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has been found (Lynch and Beare, 1990), that student IEPs had only a vague 
relationship to the activities of the students' instructional day. However, even if it is not 
linked to daily programming, it is a formalization of the student's overall instructional 
framework and a vehicle for comparing a student's status at the beginning of service 
with the student's status after receiving instructional programs related to annual goals 
and objectives. 

♦ The NAEP has been the measure of choice for assessment of students in relationship to 
achieving desired standards or performance levels on a national scale. With the push for 
reform, the NAEP has been revised to mirror the curricular emphases. 

♦ The purpose of this paper is to describe the degree to which the IEPs of students with 
disabilities match the established standards and statements of expected performance in 
the context area of mathematics, 

♦ Mathematics goals and objectives sections of the IEPs of 76 students with disabilities 
were examined within a large Southwestern urban-suburban district and a small Eastern 
rural district. 

♦ Examinations of student IEP objectives, district objectives and standards, and sample 
items from NAEP for mathematics were done within a framework of a model useful in 
describing mathematics curricula and assessments. The model allowed for categorization 
of the level of mastery expected (Bloom's revised taxonomy), the nature of material, and 
the operations required. 

♦ In general, the NAEP sample items were distributed evenly across all three dimensions 
at both g/ade 4 and grade 8. 

♦ For level of mastery, there was some emphasis on the higher order skills of 
comprehension and application. 

♦ District objectives/standards were likely to emphasize the knowledge and computation 
categories of mastery at grade 4. Student IEPs from both schools addressed almost 
exclus : r ely lower arithmetic skills on the mastery dimension (knowledge and 
computation). Less than 15% addressed application (problem solving) processes. 

♦ The IEPs also had more holes in their coverage of materials. 

♦ Although the study utilized a limited sample, the authors concluded that if any of the 
students reached all of his or her IEP goals and objectives, he or she probably would 
receive a very disappointing score on the NAPE or a test that closely matched the 
district objectives/standards. 

♦ There is evidence that the current mathematics instruction for students in general 
education does not match the recommendations of the National Council of Mathematics. 
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The NCTM Standards are the underlying framework for the NAEP mathematics exams. 
They are also the guides that States and school districts are using to revise their math 
objectives and curricula. 

♦ There is a gap between standards and instructional practices for students with disabilities 
that is likely to grown even wider in the future. The gap between these standards and 
the mathematics instruction for special education students is even wider. 

♦ Currently, the national emphasis is on .nore complex skills and concepts. Individualized 
education plans and mathematics instruction are focusing on traditional skills and do not 
include the complexities of problem solving and abstract reasoning. 

♦ Special education teachers must recognize that their students need instruction in critical 
thinking and problem solving. IEPs should be developed to both remediate basic skills 
and develop new skills that will help the student meet the standards of his or her district, 
State, and nation. 

Spande, G. & Thurlow, M. 1994. Matching state goals to a model of school completion outcomes and 
indicators. Technical Report 9. Minneapolis, MN: University of Minnesota, National Center 
on Educational Outcomes. 

♦ One of the NCEO's activities is to determine the extent to which there is correspondence 
between States' expected outcomes and the outcomes encompassed in the NCEO 
conceptual model. This document presents the results of this matching process at the 
level of school completion with seventeen States 5 lists. 

♦ In general, the matching activity indicated that there is considerable correspondence 
between domains and outcomes in the NCEO model and State outcomes. Further 
analysis identified development of creativity an outcome identified by one State, but not 
the NCEO model. 

♦ The domain-level matching indicated that each of the NCEO domains is addressed by 
at least two States. The Academic and Functional Literacy domain is included in all 16 
States' lists of expected outcomes. The other domains that are frequently covered by 
most States are Personal and Social Adjustment (14 States), Contribution and Citizenship 
(3 States), and Physical Health (12 States). Satisfaction is addressed by only two States 
and Accommodation and Adaptations by 3 States. States range from goals that matched 
two of NCEO's domains (Colorado and Louisiana), to having goals in all eight domains 
(Indiana). Most of the States had goals in at least five NCEO domains. 
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Thurlow, M. 1993. Implications of outcomes-based education for children with disabilities. Synthesis 
Report 6, Minneapolis, MN: University of Minnesota, National Center on Educational 
Outcomes. 

♦ Outcomes-based education (OBE) is the driving force beyond the movement from 
Carnegie units to specified graduation outcomes in which students receive high school 
diplomas based on outcomes, and not just completed coursework. 

♦ The underlying premise of OBE is that all children can learn and succeed and that 
schools are responsible for ensuring the success of ail students. 

♦ Instruction should not be curriculum driven (specific book or set of instructional 
materials) but rather designed to reach a certain outcome. 

♦ OBE does not promote a specific instructional strategy. Rather, it is eclectic in 
recommendations for improving instruction. 

♦ OBE has its roots in competency-based education, criterion-based measurement, and 
mastery learning. 

♦ There have been two major OBE models; e.g., John Champlin's Outcomes-Driven 
Developmental Model (ODDM), Johnson City, New York; and William Spady's High 
Success Network (HSN) Strategic Design Model. 

♦ OBM relates closely to several other aspects of educational reform; e.g., school structure 
and management, community and business involvement, assessment techniques, and 
accountability. 

♦ Performance assessment is being utilized more and more; e.g., exhibitions, portfolios, 
essays, oral presentations, etc. 

♦ In theory, OBE is very consistent with the principles of education of students with 
disabilities in that it supports the belief that all children can learn. 

♦ In some districts within Minnesota, all students within an OBE approach were expected 
to reach the same outcomes as others. Students were given the extra support necessary 
to meet these expectations (special tutoring, extended school year, and extra assistance). 

♦ In Kentucky, the 75 developed learner outcomes apply to all students, and well as the 
evaluation of progress toward these outcomes. New assessment are administered to all 
students (with a sampling plan) except for a maximum percentage of students (2%) with 
the most significant cognitive disabilities. For these few students, the assessment system 
is being modified; however, the desired outcomes remain the same as for all students. 
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♦ OBE has some positive aspects related to students with disabilities; e.g., starting from 
what you want the student to know or do and then designing the instruction to meet that 
outcome. In addition, grading/assessment students against their own criteria rather than 
against other students relates closely to the IEP philosophy, assessment for instructional 
guidance rather than just for accountability, as well as modification in curriculum 
structures to meet student diversity are also consistent with the philosophy of special 
education. 

♦ There are also OBE pitfalls; e.g.. implementation can stress unrealistic goals, high stakes 
assessment can have unintended outcomes, outcomes can be made in a manner that does 
not consider student diversity, and ignores innovative instruction. 

Thurlow, M. & Ysseldyke, J. 1993. Can "all" ever really mean ''all" in defining and assessing student 
outcomes? Synthesis Report 5. Minneapolis, MN: University of Minnesota, National Center 
on Educational Outcomes. 

♦ This paper takes the perspective than "all can really mean all" when defining student 
outcomes. 

♦ Too often, students with disabilities have been excluded in outcomes and instructional 
programs developments. 

♦ There is confusion in the use of the terms "outcomes," "indicators," and "standards.". 

♦ NCEST considers students standards to include "specification of the content (what 
students should know and be able to do) and the level of performance that students are 
expected to attain (how good is good enough). 

♦ Outcomes have been defined as the "results of education," typically including 
knowledge, skills, and attitudes. 

♦ The National Center on Educational Outcomes (NCEO) model defines six educational 
outcomes domains and two enabling outcomes domains. 

♦ NCEO has defined an indicator as "a symbolic representation of one or more 
outcomes.. that can be used in making comparisons" (Ysseidyke et al„ 1991). Indicators 
are often the statistics derived from assessment of students related to a particular 
outcome. 

♦ This report identifies several issues when considering outcomes for all students: 

• Practically, it seems to be easier not to include students with disabilities. 

• Technically, the measurement of outcomes with typical general education 
assessment tools is difficult when students with disabilities are included. Certain 
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types of accommodation and adaptations are needed to compensate for the student's 
disability. 

Legally, the Individuals with Disabilities Education Act (IDEA) is generally not 
seen as a basis for allowing testing accommodations because it was intended to 
make available public education rather than to guarantee a particular level of 
education. Section 504 of the Rehabilitation Act of 1973, however, may be more 
demanding with respect to accommodations for individuals with disabilities. The 
impact of the Americans for Disabilities Act on schools is yet uncertain because of 
the lack of legal precedents. 

Philosophically, inclusion is at the heart of the question of involving all students. 

♦ There are several ways that "all can really mean all" in defining s f . adent outcomes. For 
example, in assessment of outcomes, a score of zero can be given to any student who 
did not actually participate in the assessment in order to decrease the exclusion rates of 
students with disabilities. 

♦ It is important to critically look at the outcomes that are defined to be certain that they 
are essential and broad enough to be relevant for all students. 

♦ There are changes needed in the assessment system beyond a single standardized test to 
measure progress on outcomes. 

Ysseldyke, J. & Thurlow, M. 1994. "Students with disabilities and educational standards: 
recommendations for policy & practice." NCEO Policy Directions. Minneapolis, MN: 
University of Minnesota, National Center on Educational Outcomes. 

♦ Standards are statements of criteria against which comparisons can be made. They may 
be criteria used to measure the quantity or quality of something. 

♦ Content standards provide specific knowledge and skills that students should have and 
be able to do as a result of exposure to the curriculum in each standards-setting area. 

♦ Performance standards define how good is good enough and can be used at a variety of 
levels. 

♦ Opportunity to learn standards are sometimes referred to as delivery standards which 
define the conditions of teaching and learning. Opportunity-to-learn standards establish 
the basis for achieving high content and performance standards. 

♦ This article provides three alternative approaches to standards and merits/limitations for 
each: 

• Using the Individualized Education Program as the basis for student standards, the 
goals and objectives might be translated into relevant outcomes that match those of 
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the school district or State. The assessment component may help define acceptable 
performance. 

• A standards for group gains system could be set up. 

• A separate system of standards can be created for students in special education. 
This article also Drovides several recommendations for content standards: 
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• Identify one set of standards. 

• Individualize the standards for students receiving special education services. 

• Specify the depth and breadth of instruction for each standard. 

• Require parent/guardian approval. 

Thurlow, ML, Ysseldyke, J., & Silverstein, B. 1993. Testing accommodations for students with 
disabilities: A review of the literature. Synthesis Report 4. Minneapolis, MN: University of 
Minnesota, National Center on Educational Outcomes. 

♦ The terms "accommodation," "modification," and "adaptation" are used interchangeably 
and sometimes convey different meanings. Testing modifications are associated with 
changes made to the test format itself (e.g., Braille, audio cassette, large-print test 
versions, reader and large-type answer sheets, etc.). Testing accommodations are 
associated with changes in the testing environment (taking the test in a different setting 
and under flexible time arrangements, etc.). 

♦ A set of guidelines about acceptable accommodations does not exist based on 
comprehensive empirical research. 

♦ This report provides a review of what is currently known about testing accommodations 
and adaptations; e.g., a summary of existing literature on testing accommodations, 
adaptations, and modifications within four topic areas: policy and legal considerations, 
technical concerns, minimum competency and certification or licensure testing efforts, 
and existing standards. 



Ysseldyke, J., Thurlow, M., & Geenen, K. 1994. Implementation of alternative methods for making 
educational accountability decisions for students with disabilities. Synthesis Report 6. 
Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. 
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♦ The primary way in which national, State, and local educators currently make 
accountability decisions for students with disabilities is through child count and 
compliance monitoring which is based on the assumption that application of optimal 
inputs and processes will lead to positive outcomes for children. 



♦ Documentation of the presence of a student in a program (chi)d count) or the compliance 
of a program with regulations, however, does not tell us whether students are achieving 
the desired results. 
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♦ This document discusses five ways in which States are demonstrating that education 
works for students with disabilities: 

Administration of norm-referenced tests States are using State-constructed 
assessments, commercially available tests, and/or comparisons of the relative 
performance of students compared to those in other States based on NAEP results 
(National Assessment of Educational Progress), even though many students with 
disabilities are excluded from participation in State and national testing programs. 

• Focus on Results or Outcomes - The educational outcomes of all students may be 
measured against a set of standards. Several States have specified desired 
educational outcomes and use either norm-referenced or performance tests to make 
judgements about whether students are achieving these outcomes. Baron and Coney 
have reported that approximately 30 States now have some form of nontraditional 
test items in their statewide assessment. Of these, nine States list portfolios. 

• IEP Aggregation - Policymakers have begun to examine the possibility of 
collecting data from lEPs using a data base to document the attainment of IEP goals 
or objectives. 

• Secondary Analysis of Extant Data - This approach re-analyzes data that already 
exists such as achievement information, drop-out data or post-employment data. 

• Accreditation Program - An accreditation system can be used for reporting student 
performance and distributing rewards and sanctions to schools and school districts. 

♦ This document also discusses various issues encountered in attempting to gather 
accountability data including: 

• attitudes which influence who should be included in State and national assessments 
and the kinds of data that should be reported for those who participate; 

• commitment by people to the importance of gathering accountability data; 

• agreement on outcomes and standards; sources of responsibility for student learning 
(e.g., general or special education); 

• extent of inclusion of students with disabilities in State and national assessments; 

• deciding how to report information and what to report; 

• determining the role of the State; using sanctions and rewards; 

• asking whether it is worth it; 

• fear of accountability; 

• local control over developing and implementing education programs; 

• lack of curriculum-test alignment; 

• lack of agreement on how to measure complex skills; 

• students in no-graded classrooms; and 

• absence of utility. 

♦ Various input, process, and outcome data are also discussed that can demonstrate that 
education is working for students with disabilities. 
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Ysseldyke, J. & Thurlow, M. 1993. Self-Study guide to the development of educational outcomes and 
indicators. Minneapolis, MN: University of Minnesota, National Center on Educational 
Outcomes. 

♦ This user's guide is intended to assist in the development of a system of outcomes and 
indicators to evaluate educational results in schools, school districts, and States. 

♦ The guide provides step-by-step guidance through a process to create a system of 
outcomes and indicators, using resource tools developed at NCEO with four steps that 
lead to a system of outcomes and indicators. It uses resource tools developed at NCEO 
that lead to a system of outcomes and indicators by establishing a solid foundation; 
developing, adopting, or adapting a model; establishing a data collection and reporting 
system; and installing the system. 

♦ State Education Agencies (SEAs) have begun to use the NCEO model of outcomes and 
indicators in a number of ways. For example, in one State, SEA personnel collect data 
on the extent to which students accomplish instructional objectives specified in their 
individualized programs (IEP). The SEA expects to develop a report on educational 
accomplishments for students with disabilities following the outline of domains in the 
NCEO model. 

♦ In another State, educators use the NCEO model to develop transition IEPs for students' 
with disabilities, requiring transition teams to develop objectives in each of the eight 
outcome domain areas in the NCEO model. 

♦ In another State, educational personnel group IEP objectives on the basis of the NCEO 
model and then use curriculum-based measurement approaches to monitor progress 
toward those objectives. 

♦ Another SEA holds a statewide competition among its school districts. The winning 
school districts produce report cards of the educational progress of students with 
disabilities based on the NCEO model. 

Ysseldyke, J. & Thurlow, M. (Eds). 1993. Views on inclusion and testing accommodations for students 
with disabilities. Synthesis Report 7. Minneapolis, MN: University of Minnesota, National 
Center on Educational Outcomes. 

♦ This document includes a set of contributed papers regarding inclusion/exclusion and 
accommodation issues. 

♦ Bob Algozzine, University of North Carolina-Charlotte, stated that excluding any student 
from testing violates the spirit and practice of inclusion. He also opposed the practice 
of substituting progress on an IEP for State and national testing results. Accommodations 
or modifications offered to one students should also be available to all students. 



Evaluation Policy Forum Background Paper Page 25 

^ Project FORUM at NASDSE q A Final Report - August 11. 1994 

ERIC u 



ir'rwj 



♦ Paul Koehler, Arizona Department of Education, indicated that removing students in 
special education from State and national accountability testing results in removal from 
the "curriculum track" since IEP goals are often unrelated to regular education goals. 
The Arizona Student Assessment Program has tried to include nearly all students with 
IEPs by developing modified (mediated) forms of the assessments. Arizona guidelines 
on "mediated assessment" provide accommodations that mirror, as closely as possible, 
"normal" instructional adaptations. 

♦ Barbara Loeding and Jerry Crittenden, University of South Florida, discussed the 
technical issues of assessing educational outcomes of students who have hearing 
impairments or deafness. Accommodations and participation in State and national testing 
should depend on the student(s) primary mode of communication, prior use of an 
interpreter, and functioning level or amount of hearing loss. 

♦ Jack Merwin, University of Minnesota, argued that it is acceptable to exclude children 
with disabilities from State and national testing because of their small numbers. In 
addition, excluding students with disabilities affect group averages less than excluding 
other subgroups (e.g., low socioeconomic status). 

♦ Dan Reschly, Iowa State University, indicated that high stakes testing leads to 
unwarranted exclusion of children with disabilities, using subtle methods of exclusion. 
He also explored the advantages and disadvantages of three inclusion/exclusion policy 
alternatives for high stakes State and national assessment (full exclusion, full inclusion, 
allowing a 2% rate of exclusion of students). The implementation of liberal 
accommodations policies would increase the perception of fairness and credibility of 
assessment programs. 

♦ Maynard Reynolds, University of Minnesota, argued for fully inclusive assessment 
practices in the domains of language, mathematics, social skills, and self-dependence, 
with acceptable test results gathered from 95% of the pupils. The remaining 5% of the 
students should be assessed through alternative testing, adaptive measurement 
procedures, or teacher judgments. 

Ysseldyke, J. & Thurlow, M. 1993. A conceptual model of educational outcomes for children and youth 
with disabilities. Working paper 2. Minneapolis, MN: University of Minnesota, National 
Center Educational Outcomes. 

♦ This paper provides a summary of a conceptual model of educational outcomes for 
students with disabilities which has evolved over time through input from many 
individuals and groups. 

♦ Educational outcomes are defined as the result of interactions between individuals and 
schooling experiences. An enabling outcome is defined as the result of interactions 
between individuals and life experiences that provide the individual with the opportunity 
to reach educational outcomes. 
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♦ Two enabling outcome domains are included - Presence/Participation (the extent to 
which an individual is located in a particular setting and the extent to which actual 
participation occurs) and Accommodation/Adaptation/Compensation (modifications in 
performance that must be made to adjust to or make up for some type of disability). 

♦ Six domains of educational outcomes are included within this conceptual model: 
Literacy, Contribution, Physical/Mental Health, Independence/ Responsibility, 
Social/behavioral Skills, and Satisfaction. 

♦ Indicators of the various outcome domains were also provided. 

♦ This document provided several fundamental assumptions guiding the ongoing activities 
ofNCEO: 

• A model of outcomes is needed for all students. At the broadest level, it should 
apply to all students regardless of the characteristics of individuals (including both 
intended and unintended outcomes; as well as direct and indirect outcomes). 

• Indicators of outcomes for students receiving special education services should be 
related, conceptually and statistically, to those identified for students without 
disabilities. 

• Indicators should be unbiased with respect to gender, culture, race, and other 
characteristics of the diversity of students in today's school population. 

• While indicators should meet research standards, those that do not could still be 
used. 

• A comprehensive system of indicators should provide data needed to make policy 
decisions at the State and national levels. 

Ysseldyke, J.,Thurlow, M., & Oilman, C. 1993. Educational outcomes and indicators for early childhood 
(Age 6). Minneapolis, MN: University of Minnesota, National Center on Educational 
Outcomes. 

♦ Outcomes and lists of possible indicators for school completion outcomes for early 
childhood (age 6), which were identified through the consensus-building process, are 
provided within the following eight outcome domains: (1) Presence and Participation; 
(2) Accommodation and Adaptation,; (3) Physical Health,; (4) Responsibility and 
Independence; (5) Contribution and Citizenship; (6) Academic and Functional Literacy; 
(7) Personal and Social Adjustment; and (8) Satisfaction. 

♦ Sample sources of data for the Contribution and Citizenship outcome domain are also 
provided. 
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Ysseldyke, J., Thurlow, M. & Gilman, C. 1993. Educational outcomes and indicators for early 
childhood (Age 3). Minneapolis, MN: University of Minnesota, National Center on 
Educational Outcomes. 

♦ Outcomes and lists of possible indicators for school completion outcomes for early 
childhood (age 3), which were identified through the consensus-building process, are 
provided within the following eight outcome domains: (1) Presence and Participation; 
(2) Accommodation and Adaptation; (3) Physical Health; (4) Responsibility and 
Independence; (5) Contribution and Citizenship; (6) Academic and Functional Literacy; 
(7) Personal and Social Adjustment; and (8) Satisfaction. 

♦ Sample sources of data for the Presence and Participation outcome domain are also 
presented. 

Ysseldyke, J., Thurlow, M., & Gilman, C. 1993. Educational outcomes and indicators for individuals 
at the post-school level. Minneapolis, MN: University of Minnesota, National Center on 
Educational Outcomes. 

♦ Outcomes and lists of possible indicators for school completion outcomes for individuals 
at the Post-School level, which were identified through the consensus-building process, 
are provided within the following eight outcome domains: (1) Presence and 

. Participation; (2) Accommodation and Adaptation; (3) Physical Health,; (4) 
Responsibility and Independence; (5) Contribution and Citizenship; (6) Academic and 
Functional Literacy; (7) Personal and Social Adjustment; and (8) Satisfaction. 

♦ Sample sources of data for the Responsibility and Independence outcome domain are 
also presented. 

Ysseldyke, J., Thurlow, M., & Gilman, C. 1993. Educational outcomes and indicators for students 
completing school. Minneapolis, MN: University of Minnesota, National Center on 
Educational Outcomes. 

♦ Outcomes and lists of possible indicators for school completion outcomes for students 
completing high school, which were identified through the consensus-building process, 
are provided within the following eight outcome domains: (1) Presence and 
Participation; (2) Accommodation and Adaptation; (3) Physical Health; (4) 
Responsibility and Independence; (5) Contribution and Citizenship; (6) Academic and 
Functional Literacy; (7) Personal and Social Adjustment, and (8) Satisfaction. 

♦ Sample sources of data for the Contribution and Citizenship outcome domain are also 
presented. 
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Ysseldyke, J M Thurlow, M., McGrew, K., & Vanderwood, M. 1994. Making decisions about the 
inclusion of students with disabilities in large-scale assessments; Report on a working 
conference to develop guidelines on inclusion and accommodations. Minneapolis, MN: 
University of Minnesota, National Center on Educational Outcomes. 

♦ This report is a summary of a meeting held in March, 1994, to discuss inclusion of 
students with disabilities in national and State large-scale assessments. Issues addressed 
include adaptations in assessments or assessment procedures for students with 
disabilities, developing a reasonable set of inclusion and accommodation decision 
guidelines, and identifying other major technical and implementation issues. 

♦ Assumptions underlying guidelines for making inclusion and accommodation decisions 
include: accuracy and fairness, capability of providing information on students with 
disabilities, sensitivity to the needs of students with disabilities, high standards for all, 
practically and cost effectiveness issues, and consistency with assessment to students' 
instructional programs and accommodations. 

♦ Current NAEP guidelines for making inclusion/exclusion decision are problematic in the 
use of percentage of time in the mainstream setting, and reliance on the IEP team (or 
some designated person) to make decisions about "meaningful participation" in 
assessments. 

♦ Students with disabilities must be included in all reporting of results. 

♦ Rather than using a percentage of time the child spends in the mainstream as a measure, 
a better indicator would be the type of curriculum used by the students. 

♦ Students with disabilities need to be included during the item development process for 
NAEP and State assessments in order to provide test developers information regarding 
ceiling and floor effects and to identify problematic items without adaptation. 

♦ The notion of assigning zero scores to students excluded from testing was not considered 
acceptable by meeting participants. Rather, consistent with the principle of inclusion, 
students with disabilities must be actually assessed. 

♦ There are three types of students with disabilities within the assessment process: those 
who can take large-scale assessments with no accommodations, those who can be 
included \ ith adaptations/accommodations, and those who need alternative assessments. 

♦ There should be an immediate step to encourage school districts to include in NAEP 
more students with disabilities who do not need accommodations, and to permit 
modifications that should not affect the validity of the test. 

♦ There is a need to monitor exclusion of students with disabilities. 
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The group made three recommendations regarding inclusion decisions: (1) include 
students with disabilities when trying out items to identify problematic item formats and 
the need for more items at the lower end; (2) include all students with disabilities in 
taking some form of assessment, (3) and include students with disabilities in the 
reporting of results. 

Initially, the modifications that can be made include making a student more comfortable 
and secure in test setting. Other modifications that raise technical questions should be 
studied; e.g., presentation alternatives, response alternatives, setting alternatives, and 
timing/scheduling alternatives. 

In addition to conducting and reporting results of follow-up studies of currently excluded 
students and removing incentives for exclusion, the group recommended that a panel be 
set up to review requests for new forms of testing modifications. 



Conclusion 

The above annotated bibliography of selected documents and reports related to special 
education program evaluation is intended to provide background information to stimulate 
discussion for participants attending the September Program Evaluation Forum. The full text of 
these documents and reports is significantly richer than the summaries provided. The reader is 
referred to the actual documents and reports for additional information. 
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